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Abstract Body 



Background/context: 

Recent early childhood intervention studies have targeted the improvement of children’s 
school readiness, where randomized intervention programs have successfully enhanced early 
socioemotional and academic competence among low-income children. The Head Start REDI 
(Research-based, Developmentally Informed) program, for instance, involves language- and 
literacy-specific components (e.g., interactive reading, phonological awareness games, and print 
activities), as well as techniques to improve children’s control of impulsive behavior and 
organization of goal-directed activity (Biennan et ah, 2008). Additionally, the Tools of the Mind 
program has successfully improved children’s cognitive self-regulation (i.e., executive 
functioning) (Diamond, Barnett, Thomas, & Munro, 2007). Also, the Chicago School Readiness 
Project (CSRP), a multicomponent mental health intervention that included teacher training and 
coaching on behavior management strategies, improved academic skills via increases in self- 
regulation (i.e., executive functioning, attention and impulse control) among preschoolers 
attending Head Start programs (Raver et ah, 2011). But what happens when children leave 
randomly assigned and highly controlled intervention contexts for “real world” early elementary 
classroom contexts where there is tremendous variance in classroom and school quality? 

Purpose / objective / research question / focus of study: 

In our poster presentation, we will examine the impact of CSRP on students’ academic 
achievement in elementary school. First, we will provide upper- and lower-bound estimates of 
the impact of CSRP on students’ academic achievement, taking into account their subsequent 
nonrandom selection into higher versus lower quality school settings. Second, we will test 
whether estimates of treatment impact vary as a function of key child (e.g., behavioral and 
poverty-related risk) and school characteristics (e.g., lower school quality in more highly 
disadvantaged neighborhoods). Third, taking a comprehensive view of children’s development, 
and controlling for treatment status, we will examine the extent to which classroom, family, and 
child characteristics, including more adaptive approaches to learning (e.g., on task behavior), 
shapes students’ academic achievement. 

Setting: 

Data were drawn from children who participated in CSRP in preschool and in follow-up 
waves occurring 1, 2, and 4 years after treatment. The CSRP preschool sites were selected on 
the basis of (a) receipt of Head Start funding, (b) having two or more classrooms that offered 
“full day” programming, and (c) location in one of seven high-poverty neighborhoods (see Raver 
et ah, 2008, for a detailed discussion of exclusionary criteria), in an effort to balance 
generalizability and feasibility. The CSRP staff completed block-by-block surveys of all seven 
neighborhoods, in which all child-serving agencies were identified and screened to detennine 
whether they met site selection criteria (including receipt of Head Start funding). Eligible sites 
were then invited to self-nominate for participation in the research project. Eighteen sites across 
seven neighborhoods completed the process and were included as CSRP sites, and two 
classrooms within each site were randomly selected for participation. Research staff successfully 
recruited 83% of the children enrolled in classrooms between Labor Day and the assigned 
enrollment cutoff date in mid-October of the school year. 
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In subsequent waves, children were followed into kindergarten, first grade, and third 
grade. Children dispersed into 94 kindergarten and 182 third grade classrooms with response 
rates of 73.6% and 77.9%, respectively. The follow-up study conducted at first grade involved 
administrative data only; no new data was collected. 

Population / Participants / Subjects: 

The original CSRP sample consists of 602 children enrolled in 35 classrooms nested 
within 18 Head Start sites. Enrolled children initially ranged from ages 3 to 5 (29% identified as 
Latino/a, and 64% identified as African American). 

Across waves, the age of children spanned from 4 to 8 years old, with the average age 
during intervention year being 4 years old, and the most recent follow up being at 4 years post- 
treatment. Preliminary analyses included 135 students in kindergarten and 165 in 1 st grade. All 
data for third grade are still being collected and cleaned but will be ready for analysis by January, 
2011. 

Intervention / Program / Practice: 

The principal aim of the CSRP intervention was to marshal several primary 
programmatic components to improve low-income preschool-aged children’s school readiness 
by increasing their emotional and behavioral adjustment. The first programmatic component 
emphasized workforce development, where CSRP provided teachers with 30 hours of training in 
strategies (e.g., rewarding positive behavior, redirecting negative behavior) that they could 
employ to provide their classrooms with more effective regulatory support and better 
management (Raver et ah, 2008; Webster-Stratton, Reid, & Hammond, 2001; Webster-Stratton, 
Reid, & Stoohniller, 2008). A second key component was to provide weekly “coaching” through 
classroom-based consultation provided by a mental health consultant (MHC) who supported 
teachers while they tried new techniques learned in the teacher training (Donohue, Falk, & 
Provet, 2000; Gorman-Smith, Beidel, Brown, Lochman, & Haaga, 2003). As an additional 
component, MHCs spent a significant portion of the school year conducting stress reduction 
workshops to help teachers to limit burnout. This was based on the premise that adults might 
have a difficult time implementing new strategies of building positive relationships with children 
who demonstrate especially challenging behaviors when the adults themselves may feel less well 
supported. Finally, MHCs provided direct child-focused consultation, working one-on-one with 
three to five children who exhibited the most challenging behavioral problems, with the view 
that these children might benefit from access to clinical psychological services that could be 
delivered through the Head Start setting (Perry, Dunne, McFadden, & Campbell, 2008). The 
intervention program lasted throughout the preschool year, with no additional CSRP services 
offered during the follow-up years. 

Research Design: 

To test the efficacy of this model, our research team conducted a clustered randomized 
controlled trial (RCT) in Chicago, IF, where children were embedded in classrooms, which were 
nested in sites. Through extensive collaboration with community-based Head Start programs in 
seven of Chicago’s most economically disadvantaged neighborhoods, CSRP was able to 
randomly assign nine Head Start sites to receive multi-component intervention services (and 
therefore serve as the “treatment group”) and another nine Head Start sites to receive a lower- 
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intensity package of services (including the support of a lower-cost “teacher’s aide” in the 
classroom one day a week); these latter sites were designated as “control group” classrooms. 

Though trainings were offered to all teachers randomized to treatment, not all teachers 
were able to take advantage of these sessions: Teachers attended three of the five trainings on 
average. Similarly, even though classroom visits were a main ingredient of the intervention 
package, some classrooms received as few as 2 1 visits while other classrooms received as many 
as 40 visits, with an average of 29 visits (or 128 hours of consultation) during the academic year. 
Analyses of consultants’ logs suggest that the most common social services offered to teachers 
were social support and coaching during MHCs’ classroom visits. 

Data Collection and Analysis: 

Data collection. Importantly, our study includes multiple indicators of children’s 
academic achievement. First, an important feature of our study is the inclusion of school records 
data, obtained from Chicago Public Schools, on children’s performance on direct assessments of 
various literacy skills (e.g., phonological awareness) based on the DIBELS ( Dynamic Indicators 
of Basic Early Literacy Skills). Second, teachers reported on children’s language, literacy, and 
math skills, based on measures from the Early Childhood Longitudinal Study-Kindergarten 
Cohort. Third, CSRP was also able to access students’ grades from the Chicago Public Schools 
(Mark, Kaltreider, & Campbell, 2001; Mason, Schroeter, Combs, & Washington, 1992; Miller, 
Allensworth, & Kochanek, 2002; Rhoades & Kratochwill, 1998). 

Parents completed demographic interviews, which included questions on children’s age, 
gender, and race/ethnicity, and questions on marital status and poverty-related risk. The number 
of poverty-related risks was an index based on a sum of dummy variables indicating whether 
families lived below half the federal poverty threshold, mothers held a high school degree, and 
mothers worked 10 or fewer hours per week. 

To measure children’s behavior problems, teachers completed the Behavior Problems 
Index (BPI; Zill, 1990). The original parent-report measure was adapted from several existing 
studies of children’s behavior problems and modified in minor ways, where the version used 
here was a 28-item teacher-report measure (Zill, 1990). The measure included 18 externalizing 
items (e.g., “breaks things on purpose or deliberately destroys his/her own or another’s things”) 
and 10 internalizing items (e.g., “is withdrawn, does not want to get involved with others”). We 
used a 3-point scale, ranging from 0 = not true to 2 = very/often true. Items were summed into 
externalizing and internalizing composites, with alphas of .92 and .80, respectively. Inter-rater 
reliability was adequate, with ICC values ranging from .60 - .73. Each of the BPI and SCBE 
scores were averaged across reporters that included the child’s teacher and teaching assistant. 

Additional analyses will examine the roles of students’ more positive approaches to 
learning (e.g., on task behavior) and broader classroom characteristics. To capture 
kindergarteners’ positive approaches to learning (e.g., children’s on task behavior), teachers 
answered questions from the Cooper Farran Behavior Rating Scale (CFBRS; Cooper & Farran, 
1991). We will use two components of the Classroom Assessment Scoring System (CLASS; 
LaParo, Pianta, & Stuhlman, 2004). With CSRP’s focus on teacher-child interaction, we will 
focus on the behavior management (e.g., effective methods to prevent children’s misbehavior) 
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subscale and the emotional climate subscore (a = .92), which was composed of ratings of 
positive climate (e.g., enjoyment), negative climate (e.g., harshness), and teacher sensitivity (e.g., 
responsiveness). Items were based on a global, 7-point Likert scale, with scores of 1 and 2 in the 
“low” range, scores of 3 through 5 in the “moderate” range, and scores of 6 and 7 in the “high” 
range. With 75% of observations conducted by two observers, inter-rater reliability alphas were 
.66 and .95 for behavior management and emotional climate, respectively. Covariates also 
included: number of children, number of adults, how many teachers in the classroom held a 
bachelor’s degree, experience with depressive symptoms (Kessler et ah, 2002; a= .65), and an 
index of work-related stressors (Curbow, Spratt, Ungaretti, McDonnell, & Breckler, 2000; Li- 
Grining et ah, 2010) as well. 

Preliminary data analysis. Given the nested nature of our data, academic outcomes have 
been predicted using hierarchical linear models, with child and family characteristics at level 1, 
classroom characteristics at level 2, and site characteristics, including treatment status, at level 3. 

Findings / Results: 

Tables 1 and 2 display descriptive statistics, showing variance in a range of 
characteristics throughout the treatment and follow-up years. We will capitalize on that variance 
by testing the research questions outlined above. Our preliminary analyses detect non-significant 
intervention impacts on children’s DIB ELS scores and language, literacy, and math skills. On 
one hand, additional analyses may reveal significant impacts. On the other hand, it may be that 
we did not have enough power to detect significant differences between the treatment and control 
groups, given the smaller analytic samples used in our preliminary analyses. These smaller 
samples are due to various factors such as the inclusion of limited school records data, lag time 
between cohorts, different ages at which preschoolers began participating in CSRP, attrition, and 
retention. Additional analyses may find that particular classroom characteristics and 
kindergarteners’ more adaptive approaches to learning play important roles in their later 
academic achievement, net of treatment status. 

Conclusions: 

In our discussion, we will reflect on these issues, our findings, and the complex factors 
that need to be considered when conducting rigorous, longitudinal education research on early 
intervention programs in school settings. By examining both treatment impacts and the role of 
classroom, family, and child characteristics on students’ long-tenn academic achievement, we 
hope to inform the design of future, multi-year early childhood education interventions that span 
across preschool, kindergarten, and elementary school. 
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Appendix B. Tables and Figures 



Table 1 



Descriptive Statistics on Child and Family Characteristics 



Variable 


M or %o 


SD 


Child Characteristics 


Age (months) 


49.16 


7.38 


Gender (boy) 


47.0% 




Race/Ethnicity 


African American 


66.0% 




Latino 


27.0% 




Other 


7.0% 




Externalizing Behavior Problems 


5.76 


5.77 


Internalizing Behavior Problems 


2.25 


2.46 


Family Characteristics 


Married 


21.0% 




Family Size 


4.41 


1.68 


Number of Poverty-Related Risks 


1.09 


0.99 



Note. Behavior problems were captured using teacher reports of the Behavior 
Problem Index (BPI; Zill, 1990). Number of poverty-related risks is an index based 
on a sum of dummy variables indicating whether families lived below half the 
federal poverty threshold, mothers held a high school degree, and mothers 
worked 10 or fewer hours per week 
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Descriptive Statistics 


on Academic Measures 










Beginning of 
Year 


1 year post-treatment 
Middle of Year 


End of Year 


Variable 




M SD 


M SD 


M 


SD 


DIBELS 


PSF 


N/A 


8.31 7.87 


21.87 


17.37 




NWF 


N/A 


10.41 11.95 


21.5 


18.19 




WUF 


5.53 9.18 


16.33 14.22 


29.92 


19.21 


Language & 
Literacy 




2.89 1.01 


N/A 


N/A 


Math Skills 




2.75 1.02 


N/A 


N/A 






Beginning of 
Year 


2 years post-treatment 
Middle of Year 


End of Year 


Variable 




M SD 


M SD 


M 


SD 


DIBELS 


PSF 


18.80 15.52 


26.50 18.03 


37.52 


17.03 




NWF 


19.34 14.60 


33.93 22.88 


44.63 


26.50 




WUF 


19.28 16.13 


34.51 20.44 


41.79 


23.11 






Beginning of 
Year 


4 years post-treatment* 
Middle of Year 


End of Year 


Variable 




M SD 


M SD 


M 


SD 


DIBELS 


RTF 


30.58 18.96 


38.51 29.25 


46.03 


32.10 




ORF 


54.85 32.08 


77.58 37.50 


92.48 


36.69 




WUF 


37.13 19.16 


42.69 20.87 


49.66 


24.97 


Language & 
Literacy 




4.24 0.77 


N/A 


N/A 


Math Skills 




4.14 0.86 


N/A 


N/A 


Note. The DIBELS (Dynamic Indicators of Basic Early Literacy Skills) subtests 
include phoneme segmentation fluency (PSF) which assesses phonological 
awareness, nonsense word fluency (NWF) which assesses letter-sound correspondence, 
word use fluency (WUF) which assesses vocabulary and oral language, retell fluency 
(RTF) which assesses comprehension, and oral reading fluency (ORF) which assesses 
accuracy and fluency with connected text 

^Currently, data is available at 4 years post-treatment for Cohort 1, only 
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